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Abstract 

This  paper  presents  the  results  and  lessons  learned  from  trial  usage  of  the 
Software  Engineering  Institute’s  (SEI)  Software  Process  Framework  (SPF). 

The  SPF  was  used  to  check  consistency  of  the  Cleanroom  Software 
Engineering  (CSE)  process  against  the  SEI  Capability  Maturity  Model  (CMM)  for 
Software.  The  trial  usage  was  done  for  a  STARS  (Software  Technology  for 
Adaptable,  Reliable  Systems)  supported  software  development  project, 

Improved  Mortar  Ballistic  Computer  (IMBC),  at  the  US  Army’s  Picatinny  Arsenal 
Life  Cycle  Software  Engineering  Center  (LCSEC).  The  main  conclusions 
reached  are: 

•  The  SPF  is  a  valuable  tool  for  checking  a  software  process  for  consistency 
against  the  recommendations  made  by  the  SEI  CMM. 

•  The  SPF  should  be  viewed  as  a  valuable  process  definition  aid  to  support 
refining  existing  processes  and  for  defining  new  ones,  that  both  satisfy  an 
organizations  process  requirements  and  satisfy  CMM  Key  Process  Area  (KPA) 
process  assurance  criteria. 

•  Each  SPF  KPA  description  is  made  easier  to  use  than  the  CMM  equivalent 
description  through  its  concise  representation  as  an  ETVX1  based  process 
definition  and  its  concise  summary  of  roles,  training  and  measurements 
required  to  support  the  process. 

•  The  SPF,  and  the  CMM  from  which  the  SPF  was  derived,  were  designed  to 
be  software  development  process  and  life-cycle  independent. 

Consequently,  they  do  not  provide  complete  coverage  for  all  activities 
required  to  support  an  organization's  software  development  life  cycle. 


The  CSE  process  definition  is  based  upon  a  graphical  ETVX  [Arnold  92b]  process  notation. 
ETVX  (Entry  criteria,  Tasks  performed,  Verification  conditions,  and  eXit  criteria)  process 
notation  is  geared  more  toward  activities  performed,  the  validation  of  these  activities,  and  the 
state  data/documentation  maintained. 


Software  Process  Framework 


The  SPF  was  derived  from  the  CMM  for  Software  vl  .1  [Paulk  93]  in  a  format  that 
pulls  together  the  many  references  to  process  practice  contained  within  the 
CMM.  It  should  be  recognized  that  the  SEI  CMM  is  an  invaluable  tool  for 
assessing  the  maturity  of  organization  and  project  processes,  and  for 
identifying  aspects  of  these  processes  that  meet  compliance  criteria  set  forth 
for  each  CMM  KPA.  Since  the  CMM  was  developed  to  assess  the  maturity  of  an 
organization's  software  development  practices  and  to  identify  where 
weaknesses  existed  in  an  organization's  processes,  it  should  be  recognized 
that  the  CMM  is  not  a  tool  for  defining  a  software  development  process.  It  is  a 
tool  that  identifies  required  coverage  areas  for  that  process. 

One  of  the  common  complaints  concerning  the  CMM  is  the  organization  of  the 
information  contained  within  the  document.  References  to  software  process 
practice  for  a  given  CMM  KPA  are  not  located  in  the  same  section  but  are 
dispersed  throughout  the  document.  This  makes  the  job  of  reviewing  a  defined 
process  against  the  recommendations  made  by  the  CMM  particularly  difficult 
for  an  organization  trying  to  improve  their  CMM  maturity  rating  or  trying  to  define 
a  CMM  consistent  process. 

The  SPF  allows  an  organization  to  determine  whether  their  software  process 
is  consistent  with  the  recommendations  made  in  the  CMM  in  a  much  more 
organized  fashion.  It  also  permits  the  documentation  of  areas  of  inconsistency, 
allows  for  the  evaluation  of  these  areas  of  inconsistency  as  to  whether  they  are 
really  applicable  in  a  given  situation,  and  provides  a  possible  basis  for  making 
informed  decisions  on  process  improvements. 

The  SPF  comprises  a  set  of  templates  derived  from  the  CMM  and  maps  all 
CMM  KPA  specific  recommendations  into  ETVX-based  process  definition 
tables.  The  templates  include  the  specific  text  from  the  CMM,  the 
level/page/item  reference  for  the  text  from  the  CMM,  a  check  off  box  (to  indicate 
consistency),  and  space  for  a  reference  into  the  process  under  review  where 
the  process  reviewer  indicates  the  extent  of  consistency.  The  SPF  provides 
further  clarity  by  providing  sections  for  each  KPA  that  have  CMM  specific 
recommendations  for  roles,  entry  criteria,  inputs,  activities,  outputs,  exit  criteria, 
reviews/audits,  measurements,  tools,  work  products  managed  and  controlled, 
documented  procedures,  and  training.  The  SPF  covers  all  KPAs  of  all  CMM 
levels  and  the  SEI  has  made  this  available  to  the  general  public  as  a  handbook 
[Olson  94].  A  example  is  provided  in  table  1  for  the  Activities  section  of  the  Peer 
Review  (PR)  KPA  for  CMM  level  3  excerpted  from  the  SPF. 
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Activities 

References 

Peer  reviews  are  planned,  and  the  plans  are 
documented.  (L3-97,  A1) 

Peer  review  are  performed  according  to  a  documented 
procedure.  (L3-97,  A2) 

Data  on  the  conduct  and  results  of  the  peer  reviews  are 
recorded.  (L3-99,  A3) 

Measurements  are  made  and  used  to  determine  the 
status  of  the  peer  review  activities.  (L3-99,  Ml) 

The  software  quality  assurance  group  reviews  and/or 
audits  the  activities  and  work  products  for  peer  reviews 
and  reports  the  results.  (L3-100,  VI) 

Table  1 :  Example  of  Activities  for  Peer  Review  KPA 


Background 

The  LCSEC  at  Picatinny  Arsenal  is  a  representative  DoD  Software  Support 
Activity  that  wants  to  apply  a  more  formal  approach  to  software  support.  The 
current  state  of  software  engineering  at  the  LCSEC  varies  from  project  to 
project  but  the  majority  have  not  achieved  the  desired  level  of  productivity  and 
quality.  A  major  goal  of  the  LCSEC  is  to  achieve  a  SEI  CMM  level  3  rating  by 
adopting  an  evolutionary  process  improvement  approach  to  software 
engineering. 

The  Picatinny  Arsenal  was  involved  in  a  STARS-sponsored  process  technology 
transfer  demonstration  that  has  shown  very  dramatic  results  [Sherer  94],  To 
date  the  Picatinny  Arsenal  has  realized  numerous  benefits  from  the 
demonstration  project,  the  re-engineering  of  the  Mortar  Ballistic  Computer: 

•  increased  software  development  productivity  from  a  baseline  of  121  LOC 
per  person  month  to  531  LOC  per  person  month, 

•  dramatic  lowering  of  error  rates  in  software  development  to  a  level  that 
currently  is  0.25  errors  per  KLOC, 

•  dramatic  increase  in  moral  and  job  satisfaction  of  the  engineering  staff 

•  successful  transfer  of  STARS  process  technology  that  addresses  levels  2,  3 
and  5  KPAs  into  a  CMM  level  1  organization. 

Based  upon  this  successful  demonstration,  LCSEC  management  wanted  to 
achieve  a  CMM  level  3  by  evolving  the  organization’s  existing  process 
technology.  Management  decided  that  future  re-engineering  efforts  would 
make  use  of  the  STARS-sponsored  technologies.  The  problem  was  identifying 
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just  how  well  the  STARS-sponsored  technologies  met  CMM  recommendations 
for  a  level  3  organization. 


Trial  Usage  Pilot  Goals 

The  SEI,  as  a  collaborative  member  of  the  STARS  program,  was  looking  for  a 
candidate  trial  use  pilot  for  a  new  product,  the  Software  Process  Framework. 
The  SEI  was  particularly  interested  in  trial  use  of  a  well  defined  process  above 
the  CMM  level  2  (Repeat  Level)  since  there  were  few  candidate  processes 
above  level  2.  The  Picatinny  Arsenal  project  was  using  the  Cleanroom 
Software  Engineering  (CSE)  process,  which  had  been  defined  from  work 
performed  earlier  in  collaboration  between  the  SEI  and  STARS  program.  CSE 
is  a  well  defined  and  experience  tested  process,  developed  at  IBM  over  15 
years  ago,  became  part  of  the  SEI  Process  Asset  Library  (PAL)  [Arnold  92a]  and 
addresses  the  requirements  of  several  level  2,  3  and  5  KPAs.  The  KPAs  which 
this  process  addresses  in  substantial  fashion  are  show  in  table  2  below: 


Level  2  KPAs  addressed 

RM 

SPP 

SPTO 

SQA 

Requirements  Management 

Software  Project  Planning 

Software  Project  Tracking  and  Oversight 
Software  Quality  Assurance 

Level  3  KPAs  addressed 

PR 

Peer  Review 

1C 

Intergroup  Coordination 

SPE 

Software  Product  Engineering 

ISM 

Integrated  Software  Management 

Level  5  KPAs  addressed 

DP 

Defect  Prevention 

Table  2:  Area’s  of  Substantial  CMM  KPA  Coverage  by  Cleanroom 

For  the  purposes  of  the  trial  usage,  the  SEI  provided  the  SPF  for  CMM  levels  2 
and  3,  since  levels  4  and  5  were  not  complete  at  the  time  the  trial  was  setup. 
The  objectives  included: 

•  feedback  to  the  SEI  on  the  usage  of  the  SPF  against  a  CMM  level  3  process 

•  production  of  a  consistency  check  for  the  as  defined  CSE  process  against 
the  CMM 

•  production  of  a  document  that  described  for  the  LCSEC  at  the  Picatinny 
Arsenal,  what  CMM  level  2  and  3  KPAs  were  addressed  by  CSE  and  KPAs 
that  were  not  addressed. 

This  third  objective  will  facilitate  the  planning  of  process  activities  required  to 
enhance  existing  CSE  process  components  that  do  not  completely  satisfy 
Level  2  and  3  KPA  requirements,  and  those  processes  that  need  to  be 
developed  and  interfaced  with  the  CSE  process,  as  it  is  currently  being 
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practiced  at  Picatinny.  This  work  was  not  intended  to  be  a  substitute  for  a 
Picatinny  software  capability  evaluation. 

It  should  be  noted  that  the  CSE  process  does  not  address  all  activities  of  the 
systems  development  life  cycle.  Its  coverage  overlaps  with  the  traditional 
systems  engineering  activities  of  software  architecture  development  and 
software  system  specifications,  through  to  the  final  build  and  certification  of  a 
software-intensive  system.  It  also  addresses  management  oversight,  project 
monitoring  and  control.  It  was  designed  to  support  process-driven  software 
development  and  many  successful  systems  have  been  implemented  through 
its  use.  In  the  next  section  we  will  present  an  expanded  view  of  Table  2  to 
show  the  allocation  of  SPF/CMM  KPAs  and  CSE  process  components  to  the 
IMBC  process  architecture,  and  to  provide  the  reader  a  context  in  which  to  view 
the  SEI  Software  Process  Framework. 


Software  Process  Framework  and  the  Software  Development 
Process 

From  our  experiences  on  the  IMBC  Project  at  the  US  Army's  Picatinny  Arsenal, 
we  have  found  it  difficult  to  define  a  single  process  that  addresses  all  of  the 
relevant  dimensions  of  a  project.  All  software  process  definitions  for  projects 
require  at  least  three  major  project  processes  which  are  networked  and 
cooperatively  performed.  These  three  processes,  shown  on  Figure  1,  provide 
services  to  and  support  the  work  required  by  one  another.  Lower  level 
processes  are  subsequently  defined  to  support  the  mission  of,  and  are 
encapsulated  within,  their  parent  process.  Figure  1  illustrates  the  IMBC 
process  architecture  concept,  which  is  composed  of  three  major  processes: 

1)  project  management,  2)  application  engineering  and  3)  site/project  services, 
where: 

•  The  mission  of  the  project  management  process  is  to  plan  technical  work, 
review  work  status  and  progress  and  to  control  the  software  development 
project 

•  The  mission  of  the  application  engineering  process  is  to  develop  a  software 
system  that  meets  all  stated  requirements  and  quality  objectives,  within 
specified  constraints 

•  The  mission  of  the  site/project  services  process  is  to  provide  the  project 
management  process  and  the  application  engineering  process  with 
services  and  support  to  help  meet  project  objectives  for  the  software 
product  to  be  produced,  and  the  objectives  of  the  organization  of  which  the 
project  is  a  part. 

This  simple  process  architecture  provides  a  framework  for  defining  processes 
to  support: 

1 .  project  management, 
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2.  product  development,  and 

3.  product  baseline  control,  quality  assurance,  and  project  process 
management. 

We  also  used  this  process  architecture  as  a  tool,  to  depict  where  and  how  the 
Software  Process  Framework  fits,  with  respect  to  process  planning.  To 
support  our  Software  Process  Framework  pilot  goals,  we  allocated  key  CSE 
processes  and  their  respective  SPF  KPAs  to  the  high-level  IMBC  process 
architecture  components  shown  in  Figure  1.  A  more  detailed  discussion  of  the 
IMBC  Project  process  architecture  can  be  found  in  [ETT  94]. 


Figure  1:  High-Level  IMBC  Process  Architecture 

Using  the  IMBC  process  architecture  as  an  allocation  tool  enabled  us  to  better 
understand  the  correlation  between  CSE  process  components  and  SPF  KPAs. 
There  was  not  always  a  match.  It  is  important  to  recognize  that  the  SPF  and  the 
CMM  KPA  descriptions  from  which  they  were  derived,  were  intended  to  be 
process  independent.  The  SPF  provides  guidance  as  to  what  your 
organization’s  and  project  processes  should  contain  from  a  software  process 
management/assurance  perspective,  and  not  from  an  organization's  or 
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project's  process-driven  software  development  perspective.  Thus,  it  is  perfectly 
reasonable  that  a  software  development  process,  such  as  the  CSE  process 
may  contain  processes  for  which  there  is  no  correlation  to  the  SPF,  or  visa 
versa.  The  allocation  of  SPF  KPAs  to  the  high-level  IMBC  Cooperative 
Processes  are  shown  in  Table  3. 


IMBC  Process  Component 

Cleanroom 
Process  ID 

SPF  KPAs 

Project  Management 

Level  3  -  ISM 

Project  Planning 

El,  E2,  E8 

Level  2  -  SPP 

Project  Tracking 

El,  E2,  E3,  E7, 
E9 

Level  2  -  SPTO 

Project  Control 

E2,  E3,  E8 

Level  2  -  SPTO 

Level  2  -  SSM 

Baseline  Control _ 

Requirements _ E4 _ Level  2  -  RM _ 

Software  Architecture _ E4,  El 5 _ Level  3  -  SPE,  1C,  PR 

Product/Software  Releases  E4,  E6,  E15  Level  3  -  PR 

Site/Project  Services _ 

Software  Configuration  E14,  E22  Level  2  -  SCM 

Management _ 

Baseline  Maintenance  Level  2  -  SCM 

(Products/Software  Release) _ 

Software  Quality  Assurance _ Built  In _ Level  2  -  SQA _ 

SEPG  Process  Definition  Support  Level  3  -  OPD,  OPF 

Application  Engineering _ 

S/W  System  Specification  E4,  E8,  Ell,  Level  5  -  DP 

Development  E12,  E13,  E14,  Level  3  -  PR,  1C 

_  El  5,  E20,  E21  Level  2  -  SCM,  SQA 

S/W  Release(i)  Specification  E4,  E8,  Ell,  Level  5  -  DP 

Development  El  2,  El  3,  E14,  Level  3  -  PR,  1C 

_ El 5,  E20,  E21  Level  2  -  SCM,  SQA 

S/W  Release(i)  Development  E5,  E8,  Ell,  Level  3  -  PR,  1C 

_ El  6,  El 8,  E24  Level  2  -  SCM,  SQA 

S/W  Release  (i)  Certification  E5,  E8,  Ell,  Level  5  -  DP 

E6,  E17,  E19,  Level  3 -PR,  1C 

_ E22,  E23,  E24  Level  2  -  SCM,  SQA 

S/W  System  Build  and  E22,  E6,  E8,  Level  5  -  DP 

Certification  Ell,  E23  Level  3  -  1C 

_ Level  2  -  SCM,  SQA 

^S/W^System^Ope^  E23,  E8,  Ell  Level  3  -  1C 

Table  3:  IMBC  Process  Architecture  /  SPF  KPA  Allocation 


7 


The  allocation  results  shown  in  Table  3  illustrates  the  intersection  between  the 
SPF/CMM  KPAs  and  key  processes  of  the  IMBC  process  architecture.  The 
process  definition  strategy  adopted  for  process  definition  work  on  STARS  is  to: 

1 .  Define  the  work  flow  of  activities  for  a  project,  to  define  how  the  project 
intends  to  perform  software  development  (do  business), 

2.  Define  the  project's  process  architecture,  based  on  the  allocation  of  those 
process  components  and 

3.  Map  SPF/CMM  KPAs  to  the  appropriate  components  of  the  project's  process 
architecture  to  ensure  proper  CMM  KPA  coverage  is  addressed  by  the 
project's  process  components. 

To  illustrate  the  mapping  defined  in  point  3,  El  8  is  the  process  named 
"Develop  Increment  i."  Figure  2  illustrates  the  basic  tasks  required  to  support 
this  process,  and  illustrates  an  interface  to  the  "Peer  Review  Process"  to 
support  “black  box  validation."  A  software  development  process  for  an 
organization  needs  to  be  specified  to  illustrate  how  an  organization  and  its 
projects  intend  to  develop  software  products  to  facilitate  process-driven 
software  development.  The  SPF  is  needed  to  aid  in  the  planning  of  those 
processes,  to  ensure  CMM  KPA  requirements  are  addressed. 
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Figure  2:  The  Black  Box  Specification  Process  and  its  Interface 
to  the  Peer  Review  Process. 


It  should  be  noted  that  defining  processes  which  solely  address  CMM  KPAs, 
would  not  be  sufficient  to  enable  process-driven  development.  The 
management  of  software  development  efforts  and  the  product  assessment 
and  assurance  aspects  would  be  addressed,  but  the  process  for  software 
development  would  not.  KPAs  that  are  not  directly  addressed  by  the  CSE 
process  are  software  configuration  management,  training,  subcontractor 
management,  organization  process  focus,  and  organization  process  definition. 
The  omission  of  KPA  coverage  from  the  CSE  process  does  not  reduce  its 
effectiveness  as  a  process  defined  to  support  software  systems  development. 
Further,  these  KPA  coverage  areas  can  be  added  to  the  IMBC  process 
architecture,  and  interfaced  with  other  IMBC  process  components  to  provide 
support  and  services. 

Now  that  we  have  defined  our  objectives  for  the  SPF  trial  pilot  project  and 
provided  a  context  in  which  the  SPF  can  be  viewed  and  successfully  used,  we 
will  describe  the  results  obtained  from  our  pilot  effort. 
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Results  of  the  Trial  Usage  Pilot 


The  results  of  the  trial  usage  were  generally  very  positive.  The  format  and 
layout  of  the  SPF  is  a  major  improvement  in  usability  of  the  information 
contained  within  the  CMM.  Users  familiar  with  the  CMM  are  aware  of  the 
difficulty  of  trying  to  reference  all  CMM  statements  concerning  a  specific  aspect 
of  an  area  of  interest.  The  templates,  organized  by  KPA  and  specific  areas 
within  a  KPA,  made  it  relatively  easy  to  review  a  well  organized,  defined 
process.  To  check  roles  or  entry  criteria,  for  example,  as  defined  by  a  given 
process,  required  that: 

•  one  look  up  that  specific  template  for  a  given  KPA, 

•  run  down  the  list  of  recommendations, 

•  determine  how  well  documented  the  recommendation  was  in  the  defined 
process,  and 

•  enter  the  reference  for  the  defined  process. 

Additional  notes  could  be  taken  at  this  point  that  would  document  areas  for 
process  improvement. 

Quite  frequently,  the  text  reference  in  the  SPF,  was  not  a  clear  statement  that 
was  able  to  stand  by  itself  without  the  supporting  context  from  the  CMM.  This 
required  the  checking  of  the  context  from  the  CMM  to  be  able  to  perform  a 
proper  evaluation.  While  this  was  somewhat  inconvenient,  it  was  not  a  major 
problem  because  the  SPF  templates  included  the  reference  for  the 
requirement. 

The  SPF  was  a  useful  tool  for  identifying  areas  for  improvement  in  the  process 
description,  missing  elements,  and  a  substantial  number  of  improvements  to 
the  defined  process.  This  despite  the  fact  that  the  CMM  has  a  decided 
management  and  quality  assurance  frame  of  reference  rather  than  a  software 
development  reference.  The  process  description,  on  the  other  hand,  was 
geared  towards  software  development.  These  improvements  to  the  defined 
process  include: 

•  descriptions  of  roles,  the  responsibilities  required,  including  training, 

•  changes  in  terminology  to  more  closely  reflect  standard  usage, 

•  numerous  instances  of  clarifications  to  vague  requirements  in  the  defined 
process, 

•  better  identification  of  the  activities/tasks  with  the  role  responsible,  and 

•  numerous  areas  identified  where  it  would  be  appropriate  to  add  new 
process  description  for  missing  functionality  (according  to  the  CMM). 

It  should  be  pointed  out  that  the  CSE  process  description  used  for  this  review 
is  considered  to  be  a  model  of  completeness  and  one  of  the  best  examples,  in 
the  general  literature,  of  a  well  defined  process.  The  fact  that  the  SPF  could 
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improve  such  a  well  defined  process  to  this  extent  speaks  well  for  the 
usefulness  of  the  SPF. 

A  requirement  that  became  quite  evident  from  the  very  beginning  for  the 
performance  of  this  task  was  the  decision  to  use  as  a  basis  for  this  analysis 
the  version  of  CSE  process  that  had  been  defined  for  the  SEI  Process  Asset 
Library  (PAL)  [Arnold  92a].  This  is  the  basis  for  the  current  process  in  use  at 
the  Picatinny  Arsenal  but  this  process  has  been  enhanced  and  improved 
through  usage  by  the  software  development  teams.  Improvements  to  the 
process  have  not  been  documented  in  a  formal  manner  and  thus  the  task  of 
performing  an  analysis  against  this  improved  but  undocumented  process  was 
impossible.  The  SPF  requires  the  user  to  provide  a  reference  into  the 
documented  process.  If  the  improvement  is  not  documented,  it  can  not  be 
considered  to  be  repeatable  because  there  are  dependencies  on  the  users 
that  are  not  acceptable.  For  these  cases,  the  evaluation  of  the  process  was 
marked  as  non-consistent.  Little  is  included  in  the  defined  process  concerning 
support  activities  required  such  as  training  and  coaching  activities  that  are 
used  at  the  Picatinny  Arsenal.  These  activities  have  been  recognized  as  a  very 
important  ingredient  of  the  success  there  [Sherer  94],  The  defined  process  did 
not  fit  the  actual  process  in  use  at  the  Picatinny  Arsenal  but  any  other 
interpretation  opened  a  Pandora’s  box  of  intended  process  versus  defined 
process.  It  was  felt  better  to  keep  this  box  closed  or  the  second  guessing  of 
intended  functionality  would  be  impossible  to  bound.  Current  plans  include  the 
introduction  of  a  process  definition  tool  to  address  the  task  of  keeping  the 
defined  process  updated. 


Problem  Areas 

Problems  were  experienced  with  the  mapping  of  SPF/CMM  KPAs  to  the 
components  of  the  CSE  process.  The  mappings  from  the  SPF/CMM  KPAs  to 
CSE  process  components  were  not  one  to  one,  and  the  SPF/CMM  KPAs  were 
organized  at  varying  levels  of  abstraction.  For  example,  the  level  3  "Peer 
Review  KPA"  process  capabilities  are  intended  to  support  a  low  level 
development  process,  as  opposed  to  "Integrated  Software  Management  KPA" 
process  capabilities  which  is  intended  to  support  all  project  management 
processes.  This  difference  in  KPA  coverage  and  weight  makes  the  mapping 
exercise  difficult.  Consequently  some  CSE  process  components  provided 
better  SPF/CMM  KPA  coverage  than  others.  Where  the  CSE  process 
addressed  a  different  set  of  concerns  than  the  SPF/CMM  KPA,  a  conservative 
approach  was  taken  to  evaluate  compliance.  This  approach  required 
substantial  compliance  of  the  CSE  process  component  with  the  SPF  template, 
to  assert  the  CSE  process  component  as  compliant. 

The  mapping  of  SPF/CMM  KPAs  to  CSE  process  components  helped  in  our 
compliance  evaluation,  but  the  apples  and  oranges  nature  of  the  SPF/CMM 
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KPAs  and  the  process  coverage  they  address  made  compliance  evaluation  a 
challenge  in  some  cases.  This  experience  strengthened  our  belief  in 
representing  process  components  as  objects  which  provide  and  receive 
products  and  services  from  each  other,  the  excellent  organization  of  the  CSE 
process  components  as  black  boxes  enables  the  definition  of  process 
components,  which  more  closely  addressed  SPF/CMM  KPA  coverage 
requirements,  that  would  easily  interface  with  existing  CSE  processes. 

In  some  cases  the  CSE  process  components  met  the  spirit  of  the  SPF/CMM 
KPAs,  but  not  the  letter,  as  defined  in  the  SPF/CMM  KPAs.  An  example  of  this  is 
the  fact  that  the  CMM  is  skewed  towards  defect  removal  not  defect  prevention. 
The  CSE  process  takes  a  different  approach  from  traditional  software 
development  by  emphasizing  defect  prevention  and  a  method  of  testing  that  is 
based  upon  statistical  certification  of  the  reliability  of  the  product.  There  is  a 
lack  of  testing  as  defined  by  the  CMM. 

CSE  uses  a  process  called  Certification  that  uses  a  Usage  Profile,  how  the 
software  is  used  in  actual  practice,  to  accomplish  “testing”  in  the  traditional 
sense  of  the  CMM.  Statistical  analysis  of  the  Usage  Profile,  performed  by  a 
tool,  is  used  to  generate  test  cases.  The  number  of  test  cases  generated  is 
dependent  on  the  user  entered  required  quality,  i.e.  99%  reliability  of  error  free 
code  and  99%  confidence  interval.  The  resulting  product,  if  all  test  cases  run 
error  free,  has  a  statistical  certified  reliability.  Managers  have  the  capability, 
based  upon  time  and  budget  constraints,  to  make  very  informed  decisions 
about  the  quality  of  the  delivered  product.  This  methodology  has  the  added 
value  of  removing  from  the  end  product  those  errors  that  the  end  user  is  most 
likely  to  see  in  actual  usage  and  so  the  perceived  quality  of  the  product  is  also 
higher.  In  cases  such  as  these,  the  differences  were  documented  with 
justifications  provided  for  the  difference  in  approach  that  in  the  end  had  the 
same  objective,  error  free  software  at  delivery. 

The  SPF  had  a  lot  of  redundancy  in  the  template  descriptions  for  a  given  KPA. 
One  would  find  the  same  requirement  listed  under  roles,  activities,  and  exit 
criteria  for  example.  This  tends  to  be  annoying  but  does  provide  for  more  detail 
and  assures  that  nothing  is  missed.  The  rationale  for  the  redundancy, 
provided  by  the  authors  of  the  SPF,  was  that  the  templates  were  meant  to  be  a 
stand  alone  document,  allowing  one  to  take  various  perspectives  on  the 
defined  process. 
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Software  Process  Framework  Support  for  Cleanroom  Process 
Re-engineering 

The  amount  of  effort  required  to  perform  this  analysis  for  all  level  2  and  3  KPAs 
was  fourteen  (14)  person  days.  This  included  level  3  KPAs  Peer  Review, 
Intergroup  Coordination,  Software  Product  Engineering,  and  Integrated 
Software  Management.  Level  2  KPAs  included  Requirements  Management, 
Software  Project  Planning,  Software  Project  Tracking  and  Oversight,  and 
Software  Quality  Assurance.  These  were  the  KPAs  for  which  the  defined 
process  had  a  “substantial”  impact.  The  process  model  description  is 
approximately  500  pages  in  length  and  this  was  deemed  a  rather  efficient 
review  of  the  amount  of  material  included. 

In  order  to  judge  the  impact  of  this  analysis  on  the  defined  process  and  to  get 
an  indication  of  the  difficulty  of  the  analysis,  detailed  summary  data  is 
presented  in  table  4  for  the  Peer  Review  KPA  and  table  5  for  the  Intergroup 
Cooperation  KPA.  This  data  is  representative  of  the  work  performed  where  the 
SPF  identified  areas  for  improvement  in  existing  process  components,  areas 
difficult  to  map  into  existing  process  components,  and  the  complexity  of 
interpretation  required  for  each  KPA  subsection. 

The  “Difficulty  of  Mapping”  column  refers  to  the  degree  of  similarity  between  the 
defined  process  and  a  given  KPA.  If  all  required  data  for  a  KPA  was  found 
within  a  given  CSE  sub-process,  there  are  25  defined  sub-processes  within 
the  CSE  process,  the  mapping  was  considered  easy.  If  the  data  was  spread 
across  numerous  CSE  sub-processes  then  the  mapping  was  considered  to 
be  much  more  difficult. 

The  “Complexity  of  Interpretation”  refers  to  the  difficulty  in  interpreting  the 
defined  CSE  process  against  the  CMM  KPA  recommendations.  If  the  defined 
process  was  worded  and  referenced  very  closely  to  the  CMM  KPAs  it  was 
considered  to  be  easy.  If  the  defined  process  used  a  different  method  to  solve 
the  spirit  of  the  CMM  KPA  or  had  terminology  differences  which  made  the 
interpretation  more  difficult,  it  was  considered  to  be  hard. 
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KPA 

Subsection 

Total 

Number  of 
Items  in 
SPF  KPA 

(counts) 

Identified 
Areas  of 
Improve¬ 
ment 
in  Defined 
Process 
(counts) 

Identified 
Areas  of 
Improve¬ 
ment  if 
Training 
Satisfied 
(counts) 

Difficulty  of 
Mapping 

1  (easy)  to 
5(hard) 

Complexity  of 
Interpretation 
Required 

1  (easy)  to 
5(hard) 

Roles 

13 

7 

2 

1 

2 

Entry  Criteria 

10 

4 

2 

1 

1 

Inputs 

3 

0 

0 

1 

1 

Activities 

5 

1 

1 

1 

1 

Outputs 

15 

3 

3 

1 

2 

Exit  Criteria 

27 

10 

10 

1 

1 

Reviews  & 
Audits 

6 

6 

4 

1 

1 

Work 

Products 
Managed  and 
Controlled 

Not 

Applicable 

Measurement 

2 

1 

1 

1 

1 

Documented 

Procedures 

1 

0 

0 

1 

1 

Training 

2 

2 

0 

1 

1 

Tools 

Not 

Applicable 

Totals 

84 

34 

23 

Table  4:  Software  Process  Framework  -  Peer  Review  KPA 


The  numbers  for  “Identified  Areas  of  Improvement”  on  first  look  seem  to 
indicate  that  large  numbers  of  problems  were  found  in  the  review  of  this  KPA. 
This  is  not  necessarily  the  case  however  since  the  amount  of  redundancy  in 
requirements  produces  multiple  entries  in  this  column  for  one  requirement. 
The  training  recommendation  appears  under  Roles,  Entry  Criteria,  Reviews  & 
Audits  as  well  as  Training.  Therefore,  not  being  consistent  with  the  CMM 
causes  multiple  “misses”  in  the  SPF.  The  data  in  the  “Identified  Areas  of 
Improvement  if  Training  Satisfied”  column  represents  the  effect  of  compliance 
with  the  requirement  on  training  for  peer  review  leader  and  reviewers.  The 
effect  of  compliance  with  the  one  training  requirement  would  reduce  the 
number  of  Identified  Areas  of  Improvement  from  34  to  23.  There  are  additional 
cases  within  this  same  KPA  that  would  have  a  similar,  although  some  what 
less  dramatic  impact. 


14 


The  “Difficulty  of  Mapping”  and  “Complexity  of  Interpretation”  column  for  table  4 
reflect  the  very  close  relationship  between  the  defined  CSE  Peer  Review 
process  and  the  requirements  of  the  CMM.  Table  5  shows  that  there  was  a 
much  higher  level  of  difficulty  in  the  mapping  and  interpretation.  This  was  due 
to  some  significant  problems  with  deciding  the  boundaries  between  the 
Intergroup  Coordination  and  the  Software  Product  Engineering  KPAs  as 
defined  in  the  CSE  process.  There  were  additional  problems  due  to  the  fact 
that  this  KPA  spanned  so  many  CSE  sub-processes. 


KPA 

Subsection 

Total 

Number  of 
Items  in 
SPF  KPA 

Identified 
Areas  of 
Improvement 
in  Defined 

Difficulty  of 
Mapping 

Complexity  of 
Interpretation 
Required 

(counts) 

Process 

(counts) 

1  (easy)  to 
5(hard) 

1  (easy)  to 
5(hard) 

Roles 

39 

8 

5 

4 

Entry  Criteria 

7 

5 

3 

2 

Inputs 

18 

5 

5 

3 

Activities 

21 

5 

5 

2 

Outputs 

28 

6 

5 

2 

Exit  Criteria 

73 

12 

5 

2 

Reviews  & 
Audits 

16 

3 

4 

2 

Work 

Products 

Not 

Managed  and 
Controlled 

Applicable 

Measurement 

1 

1 

1 

1 

Documented 

Procedures 

2 

0 

1 

1 

Training 

3 

3 

1 

1 

Tools 

1 

1 

1 

1 

Totals 

209 

49 

Table  5:  Software  Process  Framework  -  Intergroup  Cooperation  KPA 


The  review  of  a  defined  process  against  the  SPF  requires  a  very 
knowledgeable  reviewer.  The  better  a  defined  process  is  documented,  the 
easier  the  job  will  be.  In  any  case  these  activities  are  intense  in  nature  and 
require  a  lot  of  work,  even  with  the  SPF.  The  SPF  did,  however,  make  the  job 
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much  easier  than  trying  to  deal  with  the  CMM.  The  results  of  this  analysis  will 
be  used  to  perform  three  functions: 

•  improve  the  CSE  process  definition, 

•  provide  a  map  for  the  Picatinny  Arsenal  of  areas,  not  covered  by  CSE,  that 
will  need  to  be  addressed  in  their  evolution  towards  becoming  a  level  3 
organization  and 

•  prioritize  which  process  areas  should  be  addressed  first. 

A  word  of  caution  is  in  order  for  the  potential  user  of  the  SPF.  The  SPF  is  not 
intended  to  serve  as  a  substitute  for  software  capability  evaluation.  The  SPF  is 
primarily  useful  for: 

•  the  analysis  of  defined  software  process  to  check  consistency  with  the  CMM 

•  designing  of  software  process  so  they  are  consistent  with  the  CMM,  i.e. 
process  improvement  efforts 

•  defining  of  organizational  roles,  responsibilities,  and  scope 

•  providing  recommendations  on  requirements  for  particular  CMM  levels. 

It  is  important  to  remember  that  the  CMM  has  its  greatest  strength  when  viewed 
from  the  management  and  quality  control  perspectives  and  any  processes 
designed  from  this  perspective  will  be  weak  from  the  perspective  of  software 
engineering  concerns.  This  is  not  to  minimize  the  importance  of  the  CMM,  but  it 
is  important  to  realize  that  there  are  important  areas  that  are  not  currently 
addressed  by  the  CMM. 
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